{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BNN on Pynq\n",
    "\n",
    "This notebook covers how to use Binary Neural Networks on Pynq. \n",
    "It shows an example of handwritten character recognition using a binarized neural network composed of 4 fully connected layers with 1024 neurons each, trained on the NIST database of handwritten characters. \n",
    "In order to reproduce this notebook, you will need an external USB Camera connected to the PYNQ Board."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Import the package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "require(['notebook/js/codecell'], function(codecell) {\n",
       "  codecell.CodeCell.options_default.highlight_modes[\n",
       "      'magic_text/x-csrc'] = {'reg':[/^%%microblaze/]};\n",
       "  Jupyter.notebook.events.one('kernel_ready.Kernel', function(){\n",
       "      Jupyter.notebook.get_cells().map(function(cell){\n",
       "          if (cell.cell_type == 'code'){ cell.auto_highlight(); } }) ;\n",
       "  });\n",
       "});\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import bnn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Checking available parameters\n",
    "\n",
    "By default the following trained parameters are available for LFC network using 1 bit for weights and activation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['chars_merged', 'mnist']\n"
     ]
    }
   ],
   "source": [
    "print(bnn.available_params(bnn.NETWORK_LFCW1A1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two sets of weights are available for the LFCW1A1 network, the MNIST and one for character recognition (NIST)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Instantiate the classifier\n",
    "\n",
    "Creating a classifier will automatically download the correct bitstream onto the device and load the weights trained on the specified dataset. This example works with the LFCW1A1 to recognize handwritten characters.\n",
    "Passing a runtime attribute will allow to choose between hardware accelerated or pure software inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "hw_classifier = bnn.LfcClassifier(bnn.NETWORK_LFCW1A1,\"chars_merged\",bnn.RUNTIME_HW)\n",
    "sw_classifier = bnn.LfcClassifier(bnn.NETWORK_LFCW1A1,\"chars_merged\",bnn.RUNTIME_SW)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'd', 'e', 'f', 'g', 'h', 'n', 'q', 'r', 't']\n"
     ]
    }
   ],
   "source": [
    "print(hw_classifier.classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Load the image from the camera\n",
    "\n",
    "The image is captured from the external USB camera and stored locally. The image is then enhanced in contrast and brightness to remove background noise. \n",
    "The resulting image should show the digit on a white background."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgAAAAHACAAAAADxE48PAAAIU0lEQVR4nO3c25LbNhAAUTLl//9l5mHtlcQ7KBKYQfd5cVLlONpFYwAqyo7TILL/Wr8AtWUAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGAGcAcAYAZwBwBgBnAHAGANd1AOMwjK1fQ3Tj1PoVPOR35Xv9Am/S6wQYV/5KK3oNQCcBAnAE7Ok0gHHzb/Sp0wB0Vp8BzPb86AzY1GcAOs0A4P60fgEPcOAXcALAQQJwKGzpMAAXu0SHAaiEAcD1F4AnQJH+AlhnFhu6C8CVLtNbAK5/oc4CcP1LdRbANtNY11cA81X2A6GH+gpgxvU/1nUAOtZVACvnvEPgQE8BrNwAXP8jPQWwz8eAVR0F4CPAFf0EcLj+joA1/QTghr+kmwDmn/2fZr9qXTcB6BoDgOslAB8BLuolgDN8DFjRSQA7a+ss2NVJAHOu+ll9BOAN4LI+AjjJS8ASKgAt9fC/hx8cAJMbf4cTAK6DALwBfqODAAp4GCz0F4ADoEh/ASyZxI70TwHeAL6TfgK44N9JH8DGB4HO/W7lD+AMp8S27AGUDQAtJA/g5Ei3i025A/AR4Gu5A9DXUgdwYQD4GDCTOgB9r6cATt0AHAGfMgdQ8Ajo7XBL4gDcy3dIHMCMm/ySbgI4WH/z2JA3gKsngCfHh7QBuI73SBvAjCP+oqwBFA8AC1mXNQDdpJMASva3t4d3SQPwcyB3SRqA7mIAcDkDuHICvH6Tl4A3OQPQbVIG4BXwPikD+HR2/e1kTQcBeKR/I2EAWz8WuuBPuOmV9CBhADNXJrsF/MoXwPXF8xKwIl8AulW6AJze90oXwJxz/TvpA9B3sgWwOAGuHQkeJP8kC8CFu1uyAFaM8zeGtnlfWMr1Y+I2lnocXNyr8k+Av87PAb1LNQH219g5cEWqAI6djmA0lR/dHAEvO4eBq76QaQKcPuVHl/q0RBOg6JbnnfCkPAGUrqgJnJIngHIrCfjZ8LlMd4By4zB4HdiXZgK4Y5+R5Xl45YeCliSxMvqTfOFPS3wETMP5CEbXe0OaI+DT9Pvr2YVd3Ag9U4ZhSHME7P9Y6GtrmeMrf1qOI+BghQsvBHqT8whw894mxwQ49pNE2bvFZjQkCaDoM18eBkVSHgG7W3dyY5dIMQEKnZ0DngFDigAuzfRpGCdPgxMyHgHnNq7b+5QEE+C6aRi8FB6IH8BiAQu3tk8GuzIeAeU2ngzsIsEE+HYAvP1TrvcKxgT44TsEK8JPgFt5K1yIPgFuOgHO/uk80QN4gAfBu+ABPDIALOBN8AD0NHgAXgJiB+D6PC50AMsbwD3Ht5eAl9AB6HnMABwBvyIH8OybQBv/DprIAaiCTAHcOAD8OQH/BA6AvjR1BA5gzpvbE+IG4ACoIm4Ac/cOAC8Bf+UJQI8IGwB8Y1YTNoA5r4DPiBpAxQHAnjVBA6jxLrCGIWwAqiVmABUGwNufiD4DYgaganIE4A3gMSEDqDKTjWoYhqABqJ6IAVR/BiTfAiMGoIoCBlBrAHgJGIaIAfgmYFXxAqjHzwQMAQNwANQVLoA2uCMgfABPDgCHS7wAqm5F7r5/CRZA3SXxvwiGC2DBKf2w6AHoYbECqP0M6HyJFUDLg5h6CQgVwII79HGRAmiwCS0sUgALLs/zAgXQ+BSGXgICBbDgAKggTgBtdiA+sjgBLNReG+YZEDiAOugjIEwAAT4IghwBYQJQG1ECaLf74GdAkACW699iXYhnQJAA1ErUAOCDuZ4YATSdvezWYgQQBfASECKAGFdAphABqJ0IAbQeAOhxEyGAQHiXAAOACxBA+11HPgMCBLBEXpDa2gfQfgC8i/VqKmgfgJqKGED9EwB85jQPIMbM5RbQPIAl7mK00DqAGAPgTbgX9LDWASy1GQDYsRMvgNZgIyBcANit2EjjAGDbLaBoE6DZAKBOnmgBBMCaSsECoO7Ddv60fgEvE23zhTA23nO/S95+77/qa/9a6ml9BPz9Xk+k73korQP44fI30/oIiAR5BsSYADGAlv3FAOAM4M3vCBg5z6MGAGcAcAawDnMGGMA74HOAAWygjAAD+MAbAQYAZwBbIGeAAXzCnQEGsIkxAgxghjYCDADOAOZgI8AA4AxggfX5RD8StoHyjXECbICsvwHQGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHAGQCcAcAZAJwBwBkAnAHA/Q8EuLKwyWCnUgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=512x448 at 0x7F4FA47978>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import cv2\n",
    "from PIL import Image as PIL_Image\n",
    "from PIL import ImageEnhance\n",
    "from PIL import ImageOps\n",
    "\n",
    "\n",
    "# says we capture an image from a webcam\n",
    "cap = cv2.VideoCapture(0) \n",
    "_ , cv2_im = cap.read()\n",
    "cv2_im = cv2.cvtColor(cv2_im,cv2.COLOR_BGR2RGB)\n",
    "img = PIL_Image.fromarray(cv2_im).convert(\"L\") \n",
    "\n",
    "#original captured image\n",
    "#orig_img_path = '/home/xilinx/jupyter_notebooks/bnn/pictures/webcam_image_chars.jpg'\n",
    "#img = PIL_Image.open(orig_img_path).convert(\"L\")  \n",
    "\n",
    "#Image enhancement                  \n",
    "contr = ImageEnhance.Contrast(img)  \n",
    "img = contr.enhance(5)                                                    # The enhancement values (contrast and brightness)   \n",
    "bright = ImageEnhance.Brightness(img)                                     # depends on backgroud, external lights etc  \n",
    "img = bright.enhance(2.0)            \n",
    "\n",
    "#img = img.rotate(180)                                                     # Rotate the image (depending on camera orientation)  \n",
    "#Adding a border for future cropping  \n",
    "img = ImageOps.expand(img,border=80,fill='white')   \n",
    "threshold = 180    \n",
    "img = img.point(lambda p: p > threshold and 255)   \n",
    "img\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Crop and scale the image "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image as PIL_Image\n",
    "import numpy as np\n",
    "import math\n",
    "from scipy import misc\n",
    "\n",
    "#Find bounding box  \n",
    "inverted = ImageOps.invert(img)  \n",
    "box = inverted.getbbox()  \n",
    "img_new = img.crop(box)  \n",
    "width, height = img_new.size  \n",
    "ratio = min((28./height), (28./width))  \n",
    "background = PIL_Image.new('RGB', (28,28), (255,255,255))  \n",
    "if(height == width):  \n",
    "    img_new = img_new.resize((28,28))  \n",
    "elif(height>width):  \n",
    "    img_new = img_new.resize((int(width*ratio),28))  \n",
    "    background.paste(img_new, (int((28-img_new.size[0])/2),int((28-img_new.size[1])/2)))  \n",
    "else:  \n",
    "    img_new = img_new.resize((28, int(height*ratio)))  \n",
    "    background.paste(img_new, (int((28-img_new.size[0])/2),int((28-img_new.size[1])/2)))  \n",
    "  \n",
    "background  \n",
    "img_data=np.asarray(background)  \n",
    "img_data = img_data[:,:,0]  \n",
    "misc.imsave('/home/xilinx/img_webcam_char.png', img_data) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Convert to BNN input format\n",
    "The image is resized to comply with the MNIST standard. The image is resized at 28x28 pixels and the colors inverted. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAAAbElEQVR4nJWRSxbAIAwCSV/vf2W60Sr51WYVHTGIgBa5LS405SABlrAr/lAKJKwzNEwVEwWezozXqdJxC8CW59sdZBrC2DN96YRL8XbpU7xv+eTZn4TgI++UMpRM4ffMGDl3pbOTxpQrC9rBBz8mJA//yq/5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x7F4E70B2B0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from array import *\n",
    "from PIL import Image as PIL_Image\n",
    "from PIL import ImageOps\n",
    "img_load = PIL_Image.open('/home/xilinx/img_webcam_char.png').convert(\"L\")  \n",
    "# Convert to BNN input format  \n",
    "# The image is resized to comply with the MNIST standard. The image is resized at 28x28 pixels and the colors inverted.   \n",
    "  \n",
    "#Resize the image and invert it (white on black)  \n",
    "smallimg = ImageOps.invert(img_load)  \n",
    "smallimg = smallimg.rotate(0)  \n",
    "  \n",
    "data_image = array('B')  \n",
    "  \n",
    "pixel = smallimg.load()  \n",
    "for x in range(0,28):  \n",
    "    for y in range(0,28):  \n",
    "        if(pixel[y,x] == 255):  \n",
    "            data_image.append(255)  \n",
    "        else:  \n",
    "            data_image.append(1)  \n",
    "          \n",
    "# Setting up the header of the MNIST format file - Required as the hardware is designed for MNIST dataset         \n",
    "hexval = \"{0:#0{1}x}\".format(1,6)  \n",
    "header = array('B')  \n",
    "header.extend([0,0,8,1,0,0])  \n",
    "header.append(int('0x'+hexval[2:][:2],16))  \n",
    "header.append(int('0x'+hexval[2:][2:],16))  \n",
    "header.extend([0,0,0,28,0,0,0,28])  \n",
    "header[3] = 3 # Changing MSB for image data (0x00000803)  \n",
    "data_image = header + data_image  \n",
    "output_file = open('/home/xilinx/img_webcam_processed', 'wb')  \n",
    "data_image.tofile(output_file)  \n",
    "output_file.close()   \n",
    "smallimg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Launching BNN in hardware\n",
    "Now, as the image has MNIST format, we can pass it to the PL and perform the inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inference took 7.00 microseconds\n",
      "Classification rate: 142857.14 images per second\n",
      "Class number: 10\n",
      "Class name: A\n"
     ]
    }
   ],
   "source": [
    "class_out=hw_classifier.classify_mnist(\"/home/xilinx/img_webcam_processed\")\n",
    "print(\"Class number: {0}\".format(class_out))\n",
    "print(\"Class name: {0}\".format(hw_classifier.bnn.class_name(class_out)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Launching BNN in Software\n",
    "For comparison we can also perform classification in software"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inference took 19212.00 microseconds\n",
      "Classification rate: 52.05 images per second\n",
      "Class number: 10\n",
      "Class name: A\n"
     ]
    }
   ],
   "source": [
    "class_out=sw_classifier.classify_mnist(\"/home/xilinx/img_webcam_processed\")\n",
    "print(\"Class number: {0}\".format(class_out))\n",
    "print(\"Class name: {0}\".format(sw_classifier.bnn.class_name(class_out)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Reset the device\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pynq import Xlnk\n",
    "\n",
    "xlnk = Xlnk()\n",
    "xlnk.xlnk_reset()"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
